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1  Executive  Summary 

This  DURIP  equipment  award  was  used  to  purchase,  install,  and  bring  on-line  a  computa¬ 
tional  cluster  for  plasma  physics  simulations.  An  SGI  integrated  cluster  environment  “ICE” 
unit  was  purchased,  featuring  192  2.8  GHz  processors  (on  24  nodes)  with  384  GB  high¬ 
speed  RAM  on  an  Infiniband  backbone.  Additional  nodes  include  an  8-processor  login  node 
(32  GB  RAM)  and  a  data  server  with  a  6  TB  RAID6  Infiniband  system,  plus  “admin”  and 
“rack  leader”  nodes.  SGI  personnel  setup  the  system  on-site,  with  additional  integration 
performed  by  our  local  IT  personnel.  SGI  also  installed  cluster  control  software  and  licenses 
for  the  SUSE-based  Linux  operating  system. 

Software  was  purchased  and  installed  including  Intel  Fortran  and  C/C  PL  compiler  suites 
(debuggers,  performance  measurements,  etc.),  the  TotalView  parallel  debugger,  and  the 
MOAB/PBS  scheduling  software. 

Comparisons  were  made  amongst  several  cluster  manufacturers,  including  Cray,  IBM, 
Dell,  Silicon  Mechanics,  Rackable  Sytems,  and  SiCortex  before  deciding  on  the  SGI  ICE 
model  as  the  most  “turn-key”  system.  The  system  was  ordered  in  late  2008  and  arrived  in 
early  2009  and  was  up  and  running  routinely  with  local  users  in  a  few  weeks. 

Local  University  of  Washington  computational  groups  have  made  extensive  use  of  the 
DURIP  cluster,  running  several  different  advanced  extended  MHD/two-fluid  codes.  This 
DURIP  award  has  greatly  increased  turnaround  time  and  productivity  for  these  groups. 
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Figure  1:  Photographs  of  the  SGI  ICE  cluster.  The  24  8-processor  compute  nodes  can  be 
seen  in  the  bottom  right  (green  LEDs).  The  admin  node  console  monitor  is  open  (video 
screen)  located  in  front  of  the  login  and  RAID  server  nodes. 
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2  Examples  of  Usage 

The  two  UW  AFOSR  computational  programs,  the  “Algorithm  Development  for  the  Two- 
Fluid  Plasma”  (PI:  U.  Shumlak,  AFOSR  Grant  #  FA9550-05-1-0159)  and  the  AFOSR  STTR 
“Electrodeless  Lorentz  Force  (ELF)  Thruster”  project  (PI:  Dr.  J.  Slough,  MSNW  L.L.C., 
Redmond,  WA,  AFOSR,  STTR  Phase  II  FA9550-08-C-0036)  subcontract  to  the  University 
of  Washington  (PI:  B.  A.  Nelson)  have  made  extensive  use  of  the  computational  cluster. 
(These  projects  are  given  priority  usage.) 

Other  complementary  numerical  physics  projects,  such  as  the  UW  Plasma  Science  and 
Innovation  Center  (PSI-Center)  have  also  made  use  of  this  cluster. 

2.1  Codes  Running  on  the  DURIP  Cluster 

Three  main  codes  have  been  used  on  the  DURIP  cluster  for  AFOSR  projects  and  by  the 
PSI-Center:  WARPX,  NIMROD,  and  HiFi. 

2.1.1  The  WARPX  Code 

In  the  UW-written  WARPX  code,  the  multi-fluid  plasma  equations  are  formulated  in  diver¬ 
gence  form,  which  includes  source  terms.  The  source  terms  couple  the  fluids  to  the  elec¬ 
tromagnetic  fields,  account  for  collisions  between  the  fluids,  and  provide  sinks  and  sources 
from  atomic  reactions.  A  high-order  algorithm  is  being  developed  that  solves  the  com¬ 
plete  multi-fluid  plasma  model  such  that  equilibrium,  perturbation,  and  transient  phenom¬ 
ena  can  be  accurately  simulated.  The  algorithm  implements  the  discontinuous  Galerkin 
method  to  achieve  high-order  accuracy  and  will  use  body-fitted  computational  meshes  to 
model  realistic,  general  geometries.  Novel  numerical  approaches  are  being  investigated  to 
realize  high-order  calculations  of  complex  three-dimensional  geometries.  These  numerical 
approaches  include  efficient  time-integration  techniques,  such  as  the  Hancock  method,  and 
time-integration  methods  that  are  not  limited  by  the  shortest  timescale  of  the  system,  such 
as  implicit  backward  difference  and  spectral-deferred  correction  methods.  The  algorithm 
will  also  take  advantage  of  the  parallel  computing  architectures  that  are  available  locally, 
including  the  new  cluster,  as  well  as  those  at  the  Major  Shared  Resource  Centers  (Dell  Pow- 
erEdge,  Cray  XD1,  and  others).  The  new  algorithm  will  be  benchmarked  against  known 
analytical  results  and  experimental  data  from  the  Air  Force  Research  Laboratories  [the  field 
reversed  configuration  (FRC)  implosions  for  magnetized  target  fusion  (MTF)  at  Kirtland 
AFB  and  the  plasma  thruster  work  at  Edwards  AFB].  The  new  algorithm  will  be  able  to 
simulate  plasma  dynamics  using  more  physically  accurate  models.  Simulations  performed 
on  the  DURIP  cluster  are  shown  in  Figs.  2  &  3. 

2.1.2  The  NIMROD  Code 

The  subcontract  to  the  University  of  Washington  (PI:  B.  A.  Nelson)  for  numerical  support 
of  ELF  uses  the  Non  Ideal  Magnetohydrodynamic  with  Rotation,  Open  Discussion  code, 
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Figure  2:  WARPX  simulations  of  a  Z-pinch  using  “realistic”  parameters,  I  =  50  kA,  n  = 
1  x  1022  m-3,  B  =  1  T,  1  cm  pinch  radius  in  a  two-fluid  regime,  i.e.  about  3  Larmor  radii 
in  the  pinch,  indicating  a  small-scale  instability  when  the  system  is  perturbed.  Plots  are  at 
t= 0  and  0.15  fi s. 
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Figure  3:  WARPX  simulations  of  astrophysical  jets  in  3D  in  an  MHD  regime,  i.e.  10  Larmor 
radii  in  the  narrow  pinch,  and  an  asymmetric  kink  instability  develops.  Plots  are  normalized 
and  are  at  f=0  and  5  Alfven  transit  times. 
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NIMR0D[1,  2],  NIMROD  is  a  macroscopic  simulation  code  having  several  of  the  critical 
features  for  modeling  FRC  thrusters  such  as  ELF.  It  has  been  used  for  numerical  spheromak 
and  RFP  studies.  NIMROD’s  algorithm  solves  compressible  nonlinear  magneto- fluid  equa¬ 
tions  with  electric-field  terms  selected  to  represent  either  the  non-ideal  single-fluid  MHD 
model  or  two-fluid  models  of  magnetized  plasmas.  Solution  fields  are  marched  from  initial 
conditions  with  a  semi- implicit  algorithm  for  the  MHD  equations  [3].  The  time-centering  of 
velocity  is  staggered  from  magnetic  field,  mass  density,  and  temperature  so  that  the  integra¬ 
tion  of  wave-like  terms  is  symplectic.  A  semi-implicit  operator  is  used  to  extend  the  range 
of  numerical  stability  to  arbitrarily  large  values  of  cAt/h ,  where  c  is  the  magneto-acoustic 
wave  speed,  At  is  the  time  step,  and  h  is  the  mesh  spacing.  Temporal  truncation  errors 
for  wave  propagation  are  then  dispersive  and  not  dissipative,  which  is  important  for  sim¬ 
ulating  the  nearly  dissipation-free  conditions  encountered  in  astrophysical  and  laboratory 
plasmas.  In  addition,  the  MHD  semi- implicit  operator  is  based  on  the  linear  ideal-MHD 
energy  integral [4,  2]  for  accuracy  at  large  At- values. 

ELF  plasmas  are  produced  using  rotating  magnetic  fields  (RMF)  in  a  tapered  cylindrical 
geometry,  as  simulated  on  the  DURIP  cluster  using  the  NIMROD  code,  Fig.  4. 

2.1.3  The  HiFi/SEL  Codes 

The  HiFi  (high-order  finite  —  spectral  —  elements)  code  [5]  is  a  3D  extension  of  the  2D 
SEL  (spectral  elements)  code  ibid.  HiFi  solves  extended  MHD  equations  posed  in  flux- 
source  form  on  a  finite-element  hexahedral  grid.  Evaluation  of  mesh  generation  and  efficient 
parallel  solvers  have  been  principally  developed  on  the  DURIP  cluster. 

The  AFOSR  STTR  ELF  project  has  benefited  from  development  and  simulations  on 
the  DURIP  cluster.  Neutral  fluid  was  added  to  the  Hall  MHD  equations  in  SEL,  and 
demonstrated  transfer  of  momentum  from  ions  to  neutrals  in  a  translated  FRC,  a  key  design 
of  the  ELF  thruster.  Figure  5  shows  an  initial  FRC  equilibrium  in  a  neutral  background  that 
is  accelerated  by  external  coils.  The  compression  of  the  FRC  ionizes  the  background  neutrals, 
then  expands  and  accelerates  into  downstream  neutrals.  The  FRC  essentially  stops  as  the 
ion  momentum  is  transferred  to  the  neutrals  which  continue  moving.  This  process  could  be 
applied  continuously  to  create  a  directed  neutral  velocity  along  with  the  FRC  translation, 
providing  more  thrust. 


3  Publications  and  Presentations 

The  following  publications  are  in  preparation  that  made  extensive  use  of  the  DURIP  cluster: 

•  E.T.  Meier,  V.S.  Lukin,  U.  Shumlak,  Spectral  element  spatial  discretization  error  in 
solving  highly  anisotropic  heat  conduction  equation,  Comp.  Phys.  Comm.,  (article  in 
press)  10.1016/j.cpc.2009.12.018 

•  R.D.  Milroy,  C.C.  Kim,  and  C.R.  Sovinec,  “Magnetohydrodynamic  Simulations  of  FRC 
Formation  and  Sustainment  with  RMF  Current  Drive” ,  in  preparation 
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user:  rdm 

Sun  Nov  15  12;  17;  10  2009 


Figure  4:  NIMROD  ELF  simulation  results.  Magnetic  field  lines:  Colored  fines  are  launched 
from  z= 0.3  m  and  black  fines  are  launched  from  the  end  boundary.  A  partially  transparent 
pressure  contour  is  shown  in  the  background. 
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Figure  5:  SEL  simulations  of  ion/neutral  fluid  momentum  exchange  in  for  the  ELF  thruster. 
White/black  line  contours  are  magnetic  flux  inside/outside  the  separatrix,  pseudo-color  con¬ 
tours  are  neutral  density,  and  momentum  density,  pv,  demonstrate  transfer  from  ions  (white 
arrows)  to  neutrals  (black  arrows). 
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•  B.  Srinivasan  and  U.  Shumlak,  “A  Semi-Implicit,  Ideal,  Full  Two-Fluid  Plasma  Model” , 
in  preparation 

•  W.  Lowrie,  V.S.  Lukin,  U.  Shumlak,  “Mesh  Deformation  Metrics  Applied  to  High- 
Order  Finite  (Spectral)  Elements” ,  in  preparation 

My  2009  APS  Poster’s  work  and  ICC  2010  made  use  of  the  ICE  cluster: 

2009  APS:  ’’Plasma  Solution  Quality  in  Distorted,  Body-Fitted  Meshes  in  SEL/HiFi”, 
W.  Lowrie,  V.S.  Lukin,  U.  Shumlak. 

2010  ICC:  ’’Multi-block  capabilities  in  HiFi  and  its  application  to  modeling  the  ZaP 
z-pinch  experiment”.  W.  Lowrie,  V.S.  Lukin,  U.  Shumlak. 

The  following  conference  presentations  made  extensive  use  of  the  DURIP  cluster: 

•  R.  D.  Milroy,  C.C.  Kim,  and  C.R.  Sovinec,  “Nimrod  Simulations  of  FRC  Formation 
with  Rotating  Magnetic  Field  Current  Drive”,  poster,  APS  2009 

•  W.  Lowrie,  V.S.  Lukin,  and  U.  Shumlak,  “Plasma  Solution  Quality  in  Distorted,  Body- 
Fitted  Meshes  in  SEL/HiFi”,  poster,  APS  2009 

•  E.T.  Meier,  V.S.  Lukin,  and  U.  Shumlak,  “Two-fluid  plasma-neutral  model  develop¬ 
ment  and  application” ,  poster,  Innovative  Confinement  Concepts  Workshop  2010 

•  R.  D.  Milroy,  C.C.  Kim,  and  C.R.  Sovinec,  “Nimrod  Simulations  of  FRC  Formation 
and  Sustainment  with  Rotating  Magnetic  Field  Current  Drive” ,  invited  talk,  Innovative 
Confinement  Concepts  Workshop  2010 

•  B.  A.  Nelson,  A.H.  Glasser,  T.R.  Jarboe,  C.C.  Kim,  G.J.  Marklin,  W.  Lowrie,  E.T. 
Meier,  R.D.  Milroy,  U.  Shumlak,  C.R.  Sovinec  and  J.B.  O’Bryan,  E.  Held,  J.-Y.  Ji, 
and  V.  Lukin,  “Simulations  of  ICC  Experiments  by  the  Plasma  Science  and  Innovation 
Center” ,  invited  talk,  Innovative  Confinement  Concepts  Workshop  2010 

•  W.  Lowrie,  V.S.  Lukin,  and  U.  Shumlak,  “Multi-block  capabilities  in  HiFi  and  its 
application  to  modeling  the  ZaP  Z-pinch  experiment” ,  poster,  Innovative  Confinement 
Concepts  Workshop  2010 

4  Summary 

A  computational  cluster  for  advanced  plasma  physics  simulations  was  selected,  ordered, 
installed,  and  is  running  routinely.  It  is  gratifying  to  hear  the  users  of  the  DURIP  cluster 
comment  about  the  benefits  of  having  a  local  developmental  cluster  to  provide  fast  turn 
around  time. 

The  DURIP  proposal  included  the  following  as  “Synergistic  Benefits  to  DoD” : 


•  Cutting  Edge  Plasma  Physics  Simulations 


—  Accelerated  development  of  two-fluid  3D  codes 

—  Accelerated  and  more  in  depth  numerical  investigation  of  the  ELF  project 
—  Synergy  with  PSI-Center  and  NIMROD  Team  research 

—  Perform  research  identified  as  important  by  the  USAF  Scientific  Advisory  Board  [6] 

•  Education  (Graduate  and  Undergraduate) 

—  Both  of  the  sponsored  AFOSR  grants  and  the  PSI-Center  strongly  involve  grad¬ 
uate  and  undergraduate  students 

—  Young  scientists  will  work  on  cutting  edge  codes  and  hardware 
These  benefits  have  already  been  realized. 
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